<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Parallel computation with OpenMP and MPI</title>
<link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
<link rel="home" href="../../index.html" title="Chapter 1. Boost.Numeric.Odeint">
<link rel="up" href="../tutorial.html" title="Tutorial">
<link rel="prev" href="using_opencl_via_vexcl.html" title="Using OpenCL via VexCL">
<link rel="next" href="all_examples.html" title="All examples">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../logo.jpg"></td>
<td align="center"><a href="../../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="using_opencl_via_vexcl.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorial.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="all_examples.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section">
<div class="titlepage"><div><div><h3 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html" title="Parallel computation with OpenMP and MPI">Parallel
      computation with OpenMP and MPI</a>
</h3></div></div></div>
<div class="toc"><dl class="toc">
<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp">OpenMP</a></span></dt>
<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.mpi">MPI</a></span></dt>
<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts">Concepts</a></span></dt>
</dl></div>
<p>
        Parallelization is a key feature for modern numerical libraries due to the
        vast availability of many cores nowadays, even on Laptops. odeint currently
        supports parallelization with OpenMP and MPI, as described in the following
        sections. However, it should be made clear from the beginning that the difficulty
        of efficiently distributing ODE integration on many cores/machines lies in
        the parallelization of the system function, which is still the user's responsibility.
        Simply using a parallel odeint backend without parallelizing the system function
        will bring you almost no performance gains.
      </p>
<div class="section">
<div class="titlepage"><div><div><h4 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp" title="OpenMP">OpenMP</a>
</h4></div></div></div>
<p>
          odeint's OpenMP support is implemented as an external backend, which needs
          to be manually included. Depending on the compiler some additional flags
          may be needed, i.e. <code class="literal">-fopenmp</code> for GCC.
        </p>
<p>
</p>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">omp</span><span class="special">.</span><span class="identifier">h</span><span class="special">&gt;</span>
<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">/</span><span class="identifier">external</span><span class="special">/</span><span class="identifier">openmp</span><span class="special">/</span><span class="identifier">openmp</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
</pre>
<p>
        </p>
<p>
          In the easiest parallelization approach with OpenMP we use a standard
          <code class="computeroutput"><span class="identifier">vector</span></code> as the state type:
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">typedef</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="keyword">double</span> <span class="special">&gt;</span> <span class="identifier">state_type</span><span class="special">;</span>
</pre>
<p>
        </p>
<p>
          We initialize the state with some random data:
        </p>
<p>
</p>
<pre class="programlisting"><span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="number">131101</span><span class="special">;</span>
<span class="identifier">state_type</span> <span class="identifier">x</span><span class="special">(</span> <span class="identifier">N</span> <span class="special">);</span>
<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">uniform_real_distribution</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span> <span class="identifier">distribution</span><span class="special">(</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">2.0</span><span class="special">*</span><span class="identifier">pi</span> <span class="special">);</span>
<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">mt19937</span> <span class="identifier">engine</span><span class="special">(</span> <span class="number">0</span> <span class="special">);</span>
<span class="identifier">generate</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">begin</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">end</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span> <span class="identifier">distribution</span> <span class="special">,</span> <span class="identifier">engine</span> <span class="special">)</span> <span class="special">);</span>
</pre>
<p>
        </p>
<p>
          Now we have to configure the stepper to use the OpenMP backend. This is
          done by explicitly providing the <code class="computeroutput"><span class="identifier">openmp_range_algebra</span></code>
          as a template parameter to the stepper. This algebra requires the state
          type to be a model of Random Access Range and will be used from multiple
          threads by the algebra.
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">typedef</span> <span class="identifier">runge_kutta4</span><span class="special">&lt;</span>
                  <span class="identifier">state_type</span> <span class="special">,</span> <span class="keyword">double</span> <span class="special">,</span>
                  <span class="identifier">state_type</span> <span class="special">,</span> <span class="keyword">double</span> <span class="special">,</span>
                  <span class="identifier">openmp_range_algebra</span>
                <span class="special">&gt;</span> <span class="identifier">stepper_type</span><span class="special">;</span>
</pre>
<p>
        </p>
<p>
          Additional to providing the stepper with OpenMP parallelization we also
          need a parallelized system function to exploit the available cores. Here
          this is shown for a simple one-dimensional chain of phase oscillators with
          nearest neighbor coupling:
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">phase_chain</span>
<span class="special">{</span>
    <span class="identifier">phase_chain</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">gamma</span> <span class="special">=</span> <span class="number">0.5</span> <span class="special">)</span>
    <span class="special">:</span> <span class="identifier">m_gamma</span><span class="special">(</span> <span class="identifier">gamma</span> <span class="special">)</span> <span class="special">{</span> <span class="special">}</span>

    <span class="keyword">void</span> <span class="keyword">operator</span><span class="special">()(</span> <span class="keyword">const</span> <span class="identifier">state_type</span> <span class="special">&amp;</span><span class="identifier">x</span> <span class="special">,</span> <span class="identifier">state_type</span> <span class="special">&amp;</span><span class="identifier">dxdt</span> <span class="special">,</span> <span class="keyword">double</span> <span class="comment">/* t */</span> <span class="special">)</span> <span class="keyword">const</span>
    <span class="special">{</span>
        <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">size</span><span class="special">();</span>
        <span class="preprocessor">#pragma</span> <span class="identifier">omp</span> <span class="identifier">parallel</span> <span class="keyword">for</span> <span class="identifier">schedule</span><span class="special">(</span><span class="identifier">runtime</span><span class="special">)</span>
        <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">i</span> <span class="special">=</span> <span class="number">1</span> <span class="special">;</span> <span class="identifier">i</span> <span class="special">&lt;</span> <span class="identifier">N</span> <span class="special">-</span> <span class="number">1</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">i</span><span class="special">)</span>
        <span class="special">{</span>
            <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">+</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">)</span> <span class="special">+</span>
                      <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">);</span>
        <span class="special">}</span>
        <span class="identifier">dxdt</span><span class="special">[</span><span class="number">0</span>  <span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="number">1</span>  <span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="number">0</span>  <span class="special">]</span> <span class="special">);</span>
        <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">N</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">N</span><span class="special">-</span><span class="number">2</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">N</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">);</span>
    <span class="special">}</span>

    <span class="keyword">double</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">x</span> <span class="special">)</span> <span class="keyword">const</span>
    <span class="special">{</span>
        <span class="keyword">return</span> <span class="identifier">sin</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">-</span> <span class="identifier">m_gamma</span> <span class="special">*</span> <span class="special">(</span> <span class="number">1.0</span> <span class="special">-</span> <span class="identifier">cos</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">);</span>
    <span class="special">}</span>

    <span class="keyword">double</span> <span class="identifier">m_gamma</span><span class="special">;</span>
<span class="special">};</span>
</pre>
<p>
        </p>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
            In the OpenMP backends the system function will always be called sequentially
            from the thread used to start the integration.
          </p></td></tr>
</table></div>
<p>
          Finally, we perform the integration by using one of the integrate functions
          from odeint. As you can see, the parallelization is completely hidden in
          the stepper and the system function. OpenMP will take care of distributing
          the work among the threads and join them automatically.
        </p>
<p>
</p>
<pre class="programlisting"><span class="identifier">integrate_n_steps</span><span class="special">(</span> <span class="identifier">stepper_type</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">phase_chain</span><span class="special">(</span> <span class="number">1.2</span> <span class="special">)</span> <span class="special">,</span>
                   <span class="identifier">x</span> <span class="special">,</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">0.01</span> <span class="special">,</span> <span class="number">100</span> <span class="special">);</span>
</pre>
<p>
        </p>
<p>
          After integrating, the data can be accessed immediately and be processed
          further. Note, that you can specify the OpenMP scheduling by calling <code class="computeroutput"><span class="identifier">omp_set_schedule</span></code> in the beginning of
          your program:
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">int</span> <span class="identifier">chunk_size</span> <span class="special">=</span> <span class="identifier">N</span><span class="special">/</span><span class="identifier">omp_get_max_threads</span><span class="special">();</span>
<span class="identifier">omp_set_schedule</span><span class="special">(</span> <span class="identifier">omp_sched_static</span> <span class="special">,</span> <span class="identifier">chunk_size</span> <span class="special">);</span>
</pre>
<p>
        </p>
<p>
          See <a href="https://github.com/headmyshoulder/odeint-v2/blob/master/examples/openmp/phase_chain.cpp" target="_top">openmp/phase_chain.cpp</a> for the complete
          example.
        </p>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp.h0"></a>
          <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp.split_state"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp.split_state">Split
          state</a>
        </h6>
<p>
          For advanced cases odeint offers another approach to use OpenMP that allows
          for a more exact control of the parallelization. For example, for odd-sized
          data where OpenMP's thread boundaries don't match cache lines and hurt
          performance it might be advisable to copy the data from the continuous
          <code class="computeroutput"><span class="identifier">vector</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;</span></code>
          into separate, individually aligned, vectors. For this, odeint provides
          the <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;</span></code>
          type, essentially an alias for <code class="computeroutput"><span class="identifier">vector</span><span class="special">&lt;</span><span class="identifier">vector</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;&gt;</span></code>.
        </p>
<p>
          Here, the initialization is done with a <code class="computeroutput"><span class="identifier">vector</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span></code>, but then we use odeint's <code class="computeroutput"><span class="identifier">split</span></code> function to fill an <code class="computeroutput"><span class="identifier">openmp_state</span></code>. The splitting is done such
          that the sizes of the individual regions differ at most by 1 to make the
          computation as uniform as possible.
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="number">131101</span><span class="special">;</span>
<span class="identifier">vector</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span> <span class="identifier">x</span><span class="special">(</span> <span class="identifier">N</span> <span class="special">);</span>
<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">uniform_real_distribution</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span> <span class="identifier">distribution</span><span class="special">(</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">2.0</span><span class="special">*</span><span class="identifier">pi</span> <span class="special">);</span>
<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">mt19937</span> <span class="identifier">engine</span><span class="special">(</span> <span class="number">0</span> <span class="special">);</span>
<span class="identifier">generate</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">begin</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">end</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span> <span class="identifier">distribution</span> <span class="special">,</span> <span class="identifier">engine</span> <span class="special">)</span> <span class="special">);</span>
<span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">blocks</span> <span class="special">=</span> <span class="identifier">omp_get_max_threads</span><span class="special">();</span>
<span class="identifier">state_type</span> <span class="identifier">x_split</span><span class="special">(</span> <span class="identifier">blocks</span> <span class="special">);</span>
<span class="identifier">split</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">,</span> <span class="identifier">x_split</span> <span class="special">);</span>
</pre>
<p>
        </p>
<p>
          Of course, the system function has to be changed to deal with the <code class="computeroutput"><span class="identifier">openmp_state</span></code>. Note that each sub-region
          of the state is computed in a single task, but at the borders read access
          to the neighbouring regions is required.
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">phase_chain_omp_state</span>
<span class="special">{</span>
    <span class="identifier">phase_chain_omp_state</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">gamma</span> <span class="special">=</span> <span class="number">0.5</span> <span class="special">)</span>
    <span class="special">:</span> <span class="identifier">m_gamma</span><span class="special">(</span> <span class="identifier">gamma</span> <span class="special">)</span> <span class="special">{</span> <span class="special">}</span>

    <span class="keyword">void</span> <span class="keyword">operator</span><span class="special">()(</span> <span class="keyword">const</span> <span class="identifier">state_type</span> <span class="special">&amp;</span><span class="identifier">x</span> <span class="special">,</span> <span class="identifier">state_type</span> <span class="special">&amp;</span><span class="identifier">dxdt</span> <span class="special">,</span> <span class="keyword">double</span> <span class="comment">/* t */</span> <span class="special">)</span> <span class="keyword">const</span>
    <span class="special">{</span>
        <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">size</span><span class="special">();</span>
        <span class="preprocessor">#pragma</span> <span class="identifier">omp</span> <span class="identifier">parallel</span> <span class="keyword">for</span> <span class="identifier">schedule</span><span class="special">(</span><span class="identifier">runtime</span><span class="special">)</span>
        <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">n</span> <span class="special">=</span> <span class="number">0</span> <span class="special">;</span> <span class="identifier">n</span> <span class="special">&lt;</span> <span class="identifier">N</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">n</span><span class="special">)</span>
        <span class="special">{</span>
            <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">M</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">].</span><span class="identifier">size</span><span class="special">();</span>
            <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">m</span> <span class="special">=</span> <span class="number">1</span> <span class="special">;</span> <span class="identifier">m</span> <span class="special">&lt;</span> <span class="identifier">M</span><span class="special">-</span><span class="number">1</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">m</span><span class="special">)</span>
            <span class="special">{</span>
                <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">+</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">]</span> <span class="special">)</span> <span class="special">+</span>
                             <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">]</span> <span class="special">);</span>
            <span class="special">}</span>
            <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">0</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">0</span><span class="special">]</span> <span class="special">);</span>
            <span class="keyword">if</span><span class="special">(</span> <span class="identifier">n</span> <span class="special">&gt;</span> <span class="number">0</span> <span class="special">)</span>
            <span class="special">{</span>
                <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">0</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">-</span><span class="number">1</span><span class="special">].</span><span class="identifier">back</span><span class="special">()</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">].</span><span class="identifier">front</span><span class="special">()</span> <span class="special">);</span>
            <span class="special">}</span>
            <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">2</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">);</span>
            <span class="keyword">if</span><span class="special">(</span> <span class="identifier">n</span> <span class="special">&lt;</span> <span class="identifier">N</span><span class="special">-</span><span class="number">1</span> <span class="special">)</span>
            <span class="special">{</span>
                <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">+</span><span class="number">1</span><span class="special">].</span><span class="identifier">front</span><span class="special">()</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">].</span><span class="identifier">back</span><span class="special">()</span> <span class="special">);</span>
            <span class="special">}</span>
        <span class="special">}</span>
    <span class="special">}</span>

    <span class="keyword">double</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">x</span> <span class="special">)</span> <span class="keyword">const</span>
    <span class="special">{</span>
        <span class="keyword">return</span> <span class="identifier">sin</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">-</span> <span class="identifier">m_gamma</span> <span class="special">*</span> <span class="special">(</span> <span class="number">1.0</span> <span class="special">-</span> <span class="identifier">cos</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">);</span>
    <span class="special">}</span>

    <span class="keyword">double</span> <span class="identifier">m_gamma</span><span class="special">;</span>
<span class="special">};</span>
</pre>
<p>
        </p>
<p>
          Using the <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;</span></code> state type automatically selects <code class="computeroutput"><span class="identifier">openmp_algebra</span></code> which executes odeint's
          internal computations on parallel regions. Hence, no manual configuration
          of the stepper is necessary. At the end of the integration, we use <code class="computeroutput"><span class="identifier">unsplit</span></code> to concatenate the sub-regions
          back together into a single vector.
        </p>
<p>
</p>
<pre class="programlisting"><span class="identifier">integrate_n_steps</span><span class="special">(</span> <span class="identifier">runge_kutta4</span><span class="special">&lt;</span><span class="identifier">state_type</span><span class="special">&gt;()</span> <span class="special">,</span> <span class="identifier">phase_chain_omp_state</span><span class="special">(</span> <span class="number">1.2</span> <span class="special">)</span> <span class="special">,</span>
                   <span class="identifier">x_split</span> <span class="special">,</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">0.01</span> <span class="special">,</span> <span class="number">100</span> <span class="special">);</span>
<span class="identifier">unsplit</span><span class="special">(</span> <span class="identifier">x_split</span> <span class="special">,</span> <span class="identifier">x</span> <span class="special">);</span>
</pre>
<p>
        </p>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
            You don't actually need to use <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;</span></code> for advanced use cases, <code class="computeroutput"><span class="identifier">openmp_algebra</span></code> is simply an alias for
            <code class="computeroutput"><span class="identifier">openmp_nested_algebra</span><span class="special">&lt;</span><span class="identifier">range_algebra</span><span class="special">&gt;</span></code> and supports any model of Random
            Access Range as the outer, parallel state type, and will use the given
            algebra on its elements.
          </p></td></tr>
</table></div>
<p>
          See <a href="https://github.com/headmyshoulder/odeint-v2/blob/master/examples/openmp/phase_chain_omp_state.cpp" target="_top">openmp/phase_chain_omp_state.cpp</a> for
          the complete example.
        </p>
</div>
<div class="section">
<div class="titlepage"><div><div><h4 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.mpi"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.mpi" title="MPI">MPI</a>
</h4></div></div></div>
<p>
          To expand the parallel computation across multiple machines we can use
          MPI.
        </p>
<p>
          The system function implementation is similar to the OpenMP variant with
          split data, the main difference being that while OpenMP uses a spawn/join
          model where everything not explicitly paralleled is only executed in the
          main thread, in MPI's model each node enters the <code class="computeroutput"><span class="identifier">main</span><span class="special">()</span></code> method independently, diverging based
          on its rank and synchronizing through message-passing and explicit barriers.
        </p>
<p>
          odeint's MPI support is implemented as an external backend, too. Depending
          on the MPI implementation the code might need to be compiled with i.e.
          <code class="literal">mpic++</code>.
        </p>
<p>
</p>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">/</span><span class="identifier">external</span><span class="special">/</span><span class="identifier">mpi</span><span class="special">/</span><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
</pre>
<p>
        </p>
<p>
          Instead of reading another thread's data, we asynchronously send and receive
          the relevant data from neighbouring nodes, performing some computation
          in the interim to hide the latency.
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">phase_chain_mpi_state</span>
<span class="special">{</span>
    <span class="identifier">phase_chain_mpi_state</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">gamma</span> <span class="special">=</span> <span class="number">0.5</span> <span class="special">)</span>
    <span class="special">:</span> <span class="identifier">m_gamma</span><span class="special">(</span> <span class="identifier">gamma</span> <span class="special">)</span> <span class="special">{</span> <span class="special">}</span>

    <span class="keyword">void</span> <span class="keyword">operator</span><span class="special">()(</span> <span class="keyword">const</span> <span class="identifier">state_type</span> <span class="special">&amp;</span><span class="identifier">x</span> <span class="special">,</span> <span class="identifier">state_type</span> <span class="special">&amp;</span><span class="identifier">dxdt</span> <span class="special">,</span> <span class="keyword">double</span> <span class="comment">/* t */</span> <span class="special">)</span> <span class="keyword">const</span>
    <span class="special">{</span>
        <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">M</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">size</span><span class="special">();</span>
        <span class="keyword">const</span> <span class="keyword">bool</span> <span class="identifier">have_left</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()</span> <span class="special">&gt;</span> <span class="number">0</span><span class="special">,</span>
                   <span class="identifier">have_right</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()</span> <span class="special">&lt;</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">size</span><span class="special">()-</span><span class="number">1</span><span class="special">;</span>
        <span class="keyword">double</span> <span class="identifier">x_left</span><span class="special">,</span> <span class="identifier">x_right</span><span class="special">;</span>
        <span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">request</span> <span class="identifier">r_left</span><span class="special">,</span> <span class="identifier">r_right</span><span class="special">;</span>
        <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_left</span> <span class="special">)</span>
        <span class="special">{</span>
            <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">isend</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()-</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">front</span><span class="special">()</span> <span class="special">);</span> <span class="comment">// send to x_right</span>
            <span class="identifier">r_left</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">irecv</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()-</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x_left</span> <span class="special">);</span> <span class="comment">// receive from x().back()</span>
        <span class="special">}</span>
        <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_right</span> <span class="special">)</span>
        <span class="special">{</span>
            <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">isend</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()+</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">back</span><span class="special">()</span> <span class="special">);</span> <span class="comment">// send to x_left</span>
            <span class="identifier">r_right</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">irecv</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()+</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x_right</span> <span class="special">);</span> <span class="comment">// receive from x().front()</span>
        <span class="special">}</span>
        <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">m</span> <span class="special">=</span> <span class="number">1</span> <span class="special">;</span> <span class="identifier">m</span> <span class="special">&lt;</span> <span class="identifier">M</span><span class="special">-</span><span class="number">1</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">m</span><span class="special">)</span>
        <span class="special">{</span>
            <span class="identifier">dxdt</span><span class="special">()[</span><span class="identifier">m</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">+</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">]</span> <span class="special">)</span> <span class="special">+</span>
                        <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">]</span> <span class="special">);</span>
        <span class="special">}</span>
        <span class="identifier">dxdt</span><span class="special">()[</span><span class="number">0</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="number">0</span><span class="special">]</span> <span class="special">);</span>
        <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_left</span> <span class="special">)</span>
        <span class="special">{</span>
            <span class="identifier">r_left</span><span class="special">.</span><span class="identifier">wait</span><span class="special">();</span>
            <span class="identifier">dxdt</span><span class="special">()[</span><span class="number">0</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x_left</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">front</span><span class="special">()</span> <span class="special">);</span>
        <span class="special">}</span>
        <span class="identifier">dxdt</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">2</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">);</span>
        <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_right</span> <span class="special">)</span>
        <span class="special">{</span>
            <span class="identifier">r_right</span><span class="special">.</span><span class="identifier">wait</span><span class="special">();</span>
            <span class="identifier">dxdt</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x_right</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">back</span><span class="special">()</span> <span class="special">);</span>
        <span class="special">}</span>
    <span class="special">}</span>

    <span class="keyword">double</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">x</span> <span class="special">)</span> <span class="keyword">const</span>
    <span class="special">{</span>
        <span class="keyword">return</span> <span class="identifier">sin</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">-</span> <span class="identifier">m_gamma</span> <span class="special">*</span> <span class="special">(</span> <span class="number">1.0</span> <span class="special">-</span> <span class="identifier">cos</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">);</span>
    <span class="special">}</span>

    <span class="keyword">double</span> <span class="identifier">m_gamma</span><span class="special">;</span>
<span class="special">};</span>
</pre>
<p>
        </p>
<p>
          Analogous to <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;</span></code> we use <code class="computeroutput"><span class="identifier">mpi_state</span><span class="special">&lt;</span> <span class="identifier">InnerState</span><span class="special">&lt;</span><span class="identifier">T</span><span class="special">&gt;</span> <span class="special">&gt;</span></code>,
          which automatically selects <code class="computeroutput"><span class="identifier">mpi_nested_algebra</span></code>
          and the appropriate MPI-oblivious inner algebra (since our inner state
          is a <code class="computeroutput"><span class="identifier">vector</span></code>, the inner
          algebra will be <code class="computeroutput"><span class="identifier">range_algebra</span></code>
          as in the OpenMP example).
        </p>
<p>
</p>
<pre class="programlisting"><span class="keyword">typedef</span> <span class="identifier">mpi_state</span><span class="special">&lt;</span> <span class="identifier">vector</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span> <span class="special">&gt;</span> <span class="identifier">state_type</span><span class="special">;</span>
</pre>
<p>
        </p>
<p>
          In the main program we construct a <code class="computeroutput"><span class="identifier">communicator</span></code>
          which tells us the <code class="computeroutput"><span class="identifier">size</span></code>
          of the cluster and the current node's <code class="computeroutput"><span class="identifier">rank</span></code>
          within that. We generate the input data on the master node only, avoiding
          unnecessary work on the other nodes. Instead of simply copying chunks,
          <code class="computeroutput"><span class="identifier">split</span></code> acts as a MPI collective
          function here and sends/receives regions from master to each slave. The
          input argument is ignored on the slaves, but the master node receives a
          region in its output and will participate in the computation.
        </p>
<p>
</p>
<pre class="programlisting"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">environment</span> <span class="identifier">env</span><span class="special">(</span> <span class="identifier">argc</span> <span class="special">,</span> <span class="identifier">argv</span> <span class="special">);</span>
<span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">communicator</span> <span class="identifier">world</span><span class="special">;</span>

<span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="number">131101</span><span class="special">;</span>
<span class="identifier">vector</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span> <span class="identifier">x</span><span class="special">;</span>
<span class="keyword">if</span><span class="special">(</span> <span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()</span> <span class="special">==</span> <span class="number">0</span> <span class="special">)</span>
<span class="special">{</span>
    <span class="identifier">x</span><span class="special">.</span><span class="identifier">resize</span><span class="special">(</span> <span class="identifier">N</span> <span class="special">);</span>
    <span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">uniform_real_distribution</span><span class="special">&lt;</span><span class="keyword">double</span><span class="special">&gt;</span> <span class="identifier">distribution</span><span class="special">(</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">2.0</span><span class="special">*</span><span class="identifier">pi</span> <span class="special">);</span>
    <span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">mt19937</span> <span class="identifier">engine</span><span class="special">(</span> <span class="number">0</span> <span class="special">);</span>
    <span class="identifier">generate</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">begin</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">end</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span> <span class="identifier">distribution</span> <span class="special">,</span> <span class="identifier">engine</span> <span class="special">)</span> <span class="special">);</span>
<span class="special">}</span>

<span class="identifier">state_type</span> <span class="identifier">x_split</span><span class="special">(</span> <span class="identifier">world</span> <span class="special">);</span>
<span class="identifier">split</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">,</span> <span class="identifier">x_split</span> <span class="special">);</span>
</pre>
<p>
        </p>
<p>
          Now that <code class="computeroutput"><span class="identifier">x_split</span></code> contains
          (only) the local chunk for each node, we start the integration.
        </p>
<p>
          To print the result on the master node, we send the processed data back
          using <code class="computeroutput"><span class="identifier">unsplit</span></code>.
        </p>
<p>
</p>
<pre class="programlisting"><span class="identifier">integrate_n_steps</span><span class="special">(</span> <span class="identifier">runge_kutta4</span><span class="special">&lt;</span><span class="identifier">state_type</span><span class="special">&gt;()</span> <span class="special">,</span> <span class="identifier">phase_chain_mpi_state</span><span class="special">(</span> <span class="number">1.2</span> <span class="special">)</span> <span class="special">,</span>
                   <span class="identifier">x_split</span> <span class="special">,</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">0.01</span> <span class="special">,</span> <span class="number">100</span> <span class="special">);</span>
<span class="identifier">unsplit</span><span class="special">(</span> <span class="identifier">x_split</span> <span class="special">,</span> <span class="identifier">x</span> <span class="special">);</span>
</pre>
<p>
        </p>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
            <code class="computeroutput"><span class="identifier">mpi_nested_algebra</span><span class="special">::</span><span class="identifier">for_each</span></code><em class="replaceable"><code>N</code></em>
            doesn't use any MPI constructs, it simply calls the inner algebra on
            the local chunk and the system function is not guarded by any barriers
            either, so if you don't manually place any (for example in parameter
            studies cases where the elements are completely independent) you might
            see the nodes diverging, returning from this call at different times.
          </p></td></tr>
</table></div>
<p>
          See <a href="https://github.com/headmyshoulder/odeint-v2/blob/master/examples/mpi/phase_chain.cpp" target="_top">mpi/phase_chain.cpp</a> for the complete example.
        </p>
</div>
<div class="section">
<div class="titlepage"><div><div><h4 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts" title="Concepts">Concepts</a>
</h4></div></div></div>
<div class="toc"><dl class="toc">
<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state">MPI
          State</a></span></dt>
<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state">OpenMP
          Split State</a></span></dt>
<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter">Splitter</a></span></dt>
</dl></div>
<div class="section">
<div class="titlepage"><div><div><h5 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state" title="MPI State">MPI
          State</a>
</h5></div></div></div>
<p>
            As used by <code class="computeroutput"><span class="identifier">mpi_nested_algebra</span></code>.
          </p>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.h0"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.notation"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.notation">Notation</a>
          </h6>
<div class="variablelist">
<p class="title"><b></b></p>
<dl class="variablelist">
<dt><span class="term"><code class="computeroutput"><span class="identifier">InnerState</span></code></span></dt>
<dd><p>
                  The inner state type
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">State</span></code></span></dt>
<dd><p>
                  The MPI-state type
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">state</span></code></span></dt>
<dd><p>
                  Object of type <code class="computeroutput"><span class="identifier">State</span></code>
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">world</span></code></span></dt>
<dd><p>
                  Object of type <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">communicator</span></code>
                </p></dd>
</dl>
</div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.h1"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.valid_expressions"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.valid_expressions">Valid
            Expressions</a>
          </h6>
<div class="informaltable"><table class="table">
<colgroup>
<col>
<col>
<col>
<col>
</colgroup>
<thead><tr>
<th>
                    <p>
                      Name
                    </p>
                  </th>
<th>
                    <p>
                      Expression
                    </p>
                  </th>
<th>
                    <p>
                      Type
                    </p>
                  </th>
<th>
                    <p>
                      Semantics
                    </p>
                  </th>
</tr></thead>
<tbody>
<tr>
<td>
                    <p>
                      Construct a state with a communicator
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">State</span><span class="special">(</span><span class="identifier">world</span><span class="special">)</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">State</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Constructs the State.
                    </p>
                  </td>
</tr>
<tr>
<td>
                    <p>
                      Construct a state with the default communicator
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">State</span><span class="special">()</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">State</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Constructs the State.
                    </p>
                  </td>
</tr>
<tr>
<td>
                    <p>
                      Get the current node's inner state
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">state</span><span class="special">()</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">InnerState</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Returns a (const) reference.
                    </p>
                  </td>
</tr>
<tr>
<td>
                    <p>
                      Get the communicator
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">state</span><span class="special">.</span><span class="identifier">world</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">communicator</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      See <a href="http://www.boost.org/doc/libs/release/libs/mpi/" target="_top">Boost.MPI</a>.
                    </p>
                  </td>
</tr>
</tbody>
</table></div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.h2"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.models"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.models">Models</a>
          </h6>
<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
                <code class="computeroutput"><span class="identifier">mpi_state</span><span class="special">&lt;</span><span class="identifier">InnerState</span><span class="special">&gt;</span></code>
              </li></ul></div>
</div>
<div class="section">
<div class="titlepage"><div><div><h5 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state" title="OpenMP Split State">OpenMP
          Split State</a>
</h5></div></div></div>
<p>
            As used by <code class="computeroutput"><span class="identifier">openmp_nested_algebra</span></code>,
            essentially a Random Access Container with <code class="computeroutput"><span class="identifier">ValueType</span>
            <span class="special">=</span> <span class="identifier">InnerState</span></code>.
          </p>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.h0"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.notation"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.notation">Notation</a>
          </h6>
<div class="variablelist">
<p class="title"><b></b></p>
<dl class="variablelist">
<dt><span class="term"><code class="computeroutput"><span class="identifier">InnerState</span></code></span></dt>
<dd><p>
                  The inner state type
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">State</span></code></span></dt>
<dd><p>
                  The split state type
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">state</span></code></span></dt>
<dd><p>
                  Object of type <code class="computeroutput"><span class="identifier">State</span></code>
                </p></dd>
</dl>
</div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.h1"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.valid_expressions"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.valid_expressions">Valid
            Expressions</a>
          </h6>
<div class="informaltable"><table class="table">
<colgroup>
<col>
<col>
<col>
<col>
</colgroup>
<thead><tr>
<th>
                    <p>
                      Name
                    </p>
                  </th>
<th>
                    <p>
                      Expression
                    </p>
                  </th>
<th>
                    <p>
                      Type
                    </p>
                  </th>
<th>
                    <p>
                      Semantics
                    </p>
                  </th>
</tr></thead>
<tbody>
<tr>
<td>
                    <p>
                      Construct a state for <code class="computeroutput"><span class="identifier">n</span></code>
                      chunks
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">State</span><span class="special">(</span><span class="identifier">n</span><span class="special">)</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">State</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Constructs underlying <code class="computeroutput"><span class="identifier">vector</span></code>.
                    </p>
                  </td>
</tr>
<tr>
<td>
                    <p>
                      Get a chunk
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">state</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">InnerState</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Accesses underlying <code class="computeroutput"><span class="identifier">vector</span></code>.
                    </p>
                  </td>
</tr>
<tr>
<td>
                    <p>
                      Get the number of chunks
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">state</span><span class="special">.</span><span class="identifier">size</span><span class="special">()</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">size_type</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Returns size of underlying <code class="computeroutput"><span class="identifier">vector</span></code>.
                    </p>
                  </td>
</tr>
</tbody>
</table></div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.h2"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.models"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.models">Models</a>
          </h6>
<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">
                <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special">&lt;</span><span class="identifier">ValueType</span><span class="special">&gt;</span></code>
                with <code class="computeroutput"><span class="identifier">InnerState</span> <span class="special">=</span> <span class="identifier">vector</span><span class="special">&lt;</span><span class="identifier">ValueType</span><span class="special">&gt;</span></code>
              </li></ul></div>
</div>
<div class="section">
<div class="titlepage"><div><div><h5 class="title">
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter" title="Splitter">Splitter</a>
</h5></div></div></div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.h0"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.notation"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.notation">Notation</a>
          </h6>
<div class="variablelist">
<p class="title"><b></b></p>
<dl class="variablelist">
<dt><span class="term"><code class="computeroutput"><span class="identifier">Container1</span></code></span></dt>
<dd><p>
                  The continuous-data container type
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">x</span></code></span></dt>
<dd><p>
                  Object of type <code class="computeroutput"><span class="identifier">Container1</span></code>
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">Container2</span></code></span></dt>
<dd><p>
                  The chunked-data container type
                </p></dd>
<dt><span class="term"><code class="computeroutput"><span class="identifier">y</span></code></span></dt>
<dd><p>
                  Object of type <code class="computeroutput"><span class="identifier">Container2</span></code>
                </p></dd>
</dl>
</div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.h1"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.valid_expressions"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.valid_expressions">Valid
            Expressions</a>
          </h6>
<div class="informaltable"><table class="table">
<colgroup>
<col>
<col>
<col>
<col>
</colgroup>
<thead><tr>
<th>
                    <p>
                      Name
                    </p>
                  </th>
<th>
                    <p>
                      Expression
                    </p>
                  </th>
<th>
                    <p>
                      Type
                    </p>
                  </th>
<th>
                    <p>
                      Semantics
                    </p>
                  </th>
</tr></thead>
<tbody>
<tr>
<td>
                    <p>
                      Copy chunks of input to output elements
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">split</span><span class="special">(</span><span class="identifier">x</span><span class="special">,</span>
                      <span class="identifier">y</span><span class="special">)</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="keyword">void</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Calls <code class="computeroutput"><span class="identifier">split_impl</span><span class="special">&lt;</span><span class="identifier">Container1</span><span class="special">,</span> <span class="identifier">Container2</span><span class="special">&gt;::</span><span class="identifier">split</span><span class="special">(</span><span class="identifier">x</span><span class="special">,</span> <span class="identifier">y</span><span class="special">)</span></code>, splits <code class="computeroutput"><span class="identifier">x</span></code>
                      into <code class="computeroutput"><span class="identifier">y</span><span class="special">.</span><span class="identifier">size</span><span class="special">()</span></code>
                      chunks.
                    </p>
                  </td>
</tr>
<tr>
<td>
                    <p>
                      Join chunks of input elements to output
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="identifier">unsplit</span><span class="special">(</span><span class="identifier">y</span><span class="special">,</span>
                      <span class="identifier">x</span><span class="special">)</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      <code class="computeroutput"><span class="keyword">void</span></code>
                    </p>
                  </td>
<td>
                    <p>
                      Calls <code class="computeroutput"><span class="identifier">unsplit_impl</span><span class="special">&lt;</span><span class="identifier">Container2</span><span class="special">,</span> <span class="identifier">Container1</span><span class="special">&gt;::</span><span class="identifier">unsplit</span><span class="special">(</span><span class="identifier">y</span><span class="special">,</span> <span class="identifier">x</span><span class="special">)</span></code>, assumes <code class="computeroutput"><span class="identifier">x</span></code>
                      is of the correct size <span class="emphasis"><em>σ <code class="computeroutput"><span class="identifier">y</span><span class="special">[</span><span class="identifier">i</span><span class="special">].</span><span class="identifier">size</span><span class="special">()</span></code></em></span>, does not resize
                      <code class="computeroutput"><span class="identifier">x</span></code>.
                    </p>
                  </td>
</tr>
</tbody>
</table></div>
<h6>
<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.h2"></a>
            <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.models"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.models">Models</a>
          </h6>
<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
<li class="listitem">
                defined for <code class="computeroutput"><span class="identifier">Container1</span></code>
                = <a href="http://www.boost.org/doc/libs/release/libs/range/" target="_top">Boost.Range</a>
                and <code class="computeroutput"><span class="identifier">Container2</span> <span class="special">=</span>
                <span class="identifier">openmp_state</span></code>
              </li>
<li class="listitem">
                and <code class="computeroutput"><span class="identifier">Container2</span> <span class="special">=</span>
                <span class="identifier">mpi_state</span></code>.
              </li>
</ul></div>
<p>
            To implement splitters for containers incompatible with <a href="http://www.boost.org/doc/libs/release/libs/range/" target="_top">Boost.Range</a>,
            specialize the <code class="computeroutput"><span class="identifier">split_impl</span></code>
            and <code class="computeroutput"><span class="identifier">unsplit_impl</span></code> types:
</p>
<pre class="programlisting"><span class="keyword">template</span><span class="special">&lt;</span> <span class="keyword">class</span> <span class="identifier">Container1</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Container2</span> <span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Enabler</span> <span class="special">=</span> <span class="keyword">void</span> <span class="special">&gt;</span>
<span class="keyword">struct</span> <span class="identifier">split_impl</span> <span class="special">{</span>
    <span class="keyword">static</span> <span class="keyword">void</span> <span class="identifier">split</span><span class="special">(</span> <span class="keyword">const</span> <span class="identifier">Container1</span> <span class="special">&amp;</span><span class="identifier">from</span> <span class="special">,</span> <span class="identifier">Container2</span> <span class="special">&amp;</span><span class="identifier">to</span> <span class="special">);</span>
<span class="special">};</span>

<span class="keyword">template</span><span class="special">&lt;</span> <span class="keyword">class</span> <span class="identifier">Container2</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Container1</span> <span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Enabler</span> <span class="special">=</span> <span class="keyword">void</span> <span class="special">&gt;</span>
<span class="keyword">struct</span> <span class="identifier">unsplit_impl</span> <span class="special">{</span>
    <span class="keyword">static</span> <span class="keyword">void</span> <span class="identifier">unsplit</span><span class="special">(</span> <span class="keyword">const</span> <span class="identifier">Container2</span> <span class="special">&amp;</span><span class="identifier">from</span> <span class="special">,</span> <span class="identifier">Container1</span> <span class="special">&amp;</span><span class="identifier">to</span> <span class="special">);</span>
<span class="special">};</span>
</pre>
<p>
          </p>
</div>
</div>
</div>
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
<td align="left"></td>
<td align="right"><div class="copyright-footer">Copyright © 2009-2015 Karsten Ahnert and Mario Mulansky<p>
        Distributed under the Boost Software License, Version 1.0. (See accompanying
        file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
      </p>
</div></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="using_opencl_via_vexcl.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorial.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="all_examples.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>
